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1.  Introduction 

The  Image  Recognition  Group  at  NIST  has  developed  a uniform  method  of  evaluating  the  recognition  of  optical  character  readers 
used  to  process  the  information  on  electronically  scanned  forms.  NIST  Scoring  Package  Release  1.0,  NIST  Special  Software  1 
(SS1)[1],  is  distributed  on  QD-ROM  as  a reference  implementation  of  this  uniform  method  and  was  successfully  used  in  the  First 
Census  Optical  Character  Recognition  Systems  Conference.  [2] 

As  with  any  effort  related  to  technology  development,  the  Scoring  Package  has  evolved  and  matured  over  time.  The  Scoring  Pack- 
age was  originally  proposed  in  the  draft,  “Standard  Method  for  Evaluating  the  Performance  of  Systems  Intended  to  Recognize 
Hand-printed  Characters  from  Image  Data  Scanned  from  Forms”,  which  was  submitted  to  ANSI  X.3A1.  Early  implementations 
of  the  Scoring  Package  exposed  various  shortcomings  and  contradictions  within  the  draft  standard.  A public  version  of  SSI  was 
released  in  October  of  1992  along  with  “NIST  Scoring  Package  User’s  Guide  Release  1.0”  (NIS  llK  4950). [1]  The  User’s  Guide 
describes  the  reference  implementation  in  great  detail,  but  it  does  not  address  the  theory  used  to  derive  the  implementation  itself. 
In  February  of  1993,  the  paper,  “Methods  for  Evaluating  the  Performance  of  Systems  Intended  to  Recognize  Characters  from 
Image  Data  Scanned  from  Forms”  (NISTIR  5129),  replaced  the  draft  standard.  NISTIR  5129  formalizes  the  theory  used  in  the 
Scoring  Package  and  establishes  a uniform  method  of  evaluation.  [3] 

The  Methods  Paper  outlines  four  general  steps  needed  to  assess  the  performance  of  an  automated  form  processing  system.  Choos- 
ing an  optical  character  recognition  (OCR)  application  is  the  first  step.  Once  an  application  is  selected,  the  recognition  tasks 
embodied  in  the  application  and  the  interactions  between  the  tasks  that  impact  system  performance  are  identified.  Based  on  these 
tasks  and  their  interaction,  a scoring  flow  is  derived.  Scoring  accumulators  designed  to  capture  system  performance  statistics  are 
defined  within  the  scoring  flow.  Finally,  recognition  performance  measures  that  use  the  scoring  accumulators  as  input  are  defined. 
In  order  to  formalize  these  steps,  NISTIR  5 129  introduced  a standard  nomenclature  for  accumulator  names. 

The  purpose  of  this  report  is  to  map  the  nomenclature  defined  in  the  Methods  Paper  to  the  pre-existing  User’s  Gmde.  The  scoring 
flows,  scoring  accumulators,  and  performance  measures  defined  in  NISTIR  5 129  are  presented  in  Section  2.  Section  3 doounents 
the  Scoring  Package  output  files  (summary  report  and  fact  sheet)  defined  in  NIST  4950  using  the  new  nomenclature. 

2.  Scoring  Flows,  Scoring  Accumulators,  and  Performance  Measures 

The  scoring  flows  illustrated  in  this  section  are  a flexible  framework  by  which  form  processing  systems  can  be  analyzed  and  com- 
pared. However,  these  scoring  flows  should  not  be  mistaken  as  a model  for  implementing  form  processing  systems.  Three  recog- 
nition tasks  are  identified  in  these  flows:  form  identification,  field  recognition  (character  or  icon),  and  character  recognition.  These 
tasks  in  no  way  limit  the  implementation  of  a form  processing  system  by  dictating  a presumed  set  of  algorithmic  procedmes.  In 
general,  the  first  step  to  processing  a form  requires  proper  identification  of  the  form  type.  Based  on  the  identified  form  type,  the 
fields  may  be  located  through  the  use  of  a spatial  template  and  recognized.  The  task  of  reporting  a single  response  for  an  entire 
field  is  referred  to  as  field  recognition.  If  the  field  contains  non-character  information  such  as  a box  check  mark  or  a signature,  the 
task  is  referred  to  as  icon-field  recognition,  and  the  recognition  system  is  required  only  to  determine  if  the  field  contains  informa- 
tion or  not.  If  the  field  contains  characters,  the  task  is  referred  to  as  character-field  recognition.  Character-field  recognition  is 
dependent  upon  the  results  from  the  character  recognition  task  wherein  single  system  responses,  one  for  each  character  in  a field, 
are  reported. 

Systems  have  the  potential  to  reject  the  outcomes  from  each  of  these  processing  tasks.  This  is  illustrated  in  the  following  scoring 
flows  by  variables  and  branches  labeled  (A)  if  a decision  was  accepted  and  labeled  (/?)  if  rejected.  For  example,  in  Figure  1,  a 
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system  may  choose  to  reject  the  hypothesized  form  type  assigned  to  a specific  form  image,  and  in  Figure  2,  a system  may  choose 
to  reject  the  hypothesized  classification  assigned  to  a segmented  character  image.  Rejecting  outcomes  gives  a system  the  ability 
to  flag  low  confidence  decisions  as  unknown,  so  that  they  may  be  verified  by  human  inspection.  The  scoring  flows  also  include 
scoring  outcomes  that  determine  if  the  recognition  system’s  decisions  were  correct  (O  or  incorrect  (/)  or  whether  information  was 
missed  {M)  by  the  recognition  system. 

Task  interactions  of  interest  are  those  that  impact  system  performance.  In  Figure  1,  a system  rejection  of  a form’s  identification 
results  in  aU  characters  on  the  form  being  tallied  as  missed.  In  this  case,  a decision  made  within  the  form  identification  task  influ- 
ences performance  within  the  character  recognition  task  (characters  are  missed).  In  general,  system  responses  at  subsequent  tasks 
are  analyzed  only  when  the  recognition  system’s  response  at  the  current  task  has  been  accepted  by  the  system  and  the  response  is 
correct.  For  example  in  Figure  2,  only  fields  on  forms  that  have  been  accepted  and  correcdy  identified  are  analyzed  at  the  field 
recognition  task. 

In  order  to  accumulate  performance  statistics  based  on  these  tasks  and  their  interaction,  scoring  variables  must  be  defined.  Perfor- 
mance statistics  are  accumulated  at  the  form,  character-field,  icon-field,  and  character  levels  and  are  represented  as  the  variable 
subscripts /orm,  chrfld,  icofld,  and  char  respectively.  The  form  processing  task  contributing  to  a particular  statistical  accumulator 
is  denoted  by  the  variable’s  superscript.  Statistics  accumulated  for  forms  identification  are  denoted  asfrmid,  field  recognition  as 
fldrec,  and  character  recognition  as  chrrec. 

Using  this  nomenclature,  variable  accumulators  used  to  compute  system  performance  measures  are  defined.  For  example,  the  vari- 
able can  be  used  to  represent  the  total  number  of  correctly  identified  forms  accepted  by  the  recognition  system.  Likewise, 

a variable  representing  the  total  number  of  characters  missed  dming  character  recognition  is  Several  other  accumulators 

are  also  required  for  scoring.  They  include  accumulators  such  as  the  total  number  of  forms  processed  and  the  total  num- 

ber of  reference  characters  known  to  be  on  the  forms  processed  total 

Section  2.1  illustrates  scoring  flows.  Section  2.2  defines  scoring  accumulators,  and  Section  2.3  fists  performance  measures. 

2.1  Scoring  Flows 
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22  Scoring  Accumulators 


Totals 

1. 

forms  process 

2. 

character-fields  processed 

3. 

icon-fields  processed 

4.  total 

reference  characters  on  all  forms  processed 

Form  Identification 

5- 

accepted  correctly  identified  forms 

6-  A//™" 

accepted  incorrectly  identified  forms 

7-  R%t' 

rejected  form  identifications 

Character-Fields 

8- 

missed  character-fields  due  to  accepted  incorrectly  identified  forms 

9.  AMfZf 

missed  characters  due  to  accepted  incorrectly  identified  forms 

10.  RMf;!:-^, 

missed  character-fields  due  to  rejected  form  identifications 

11.  RMfZf 

missed  characters  due  to  rejected  form  identifications 

Icon-Fields 

12. 

accepted  missed  icon-fields  due  to  accepted  incorrectly  identified  forms 

‘3-  «a4™/W 

missed  icon-fields  due  to  rejected  form  identifications 

Field  Recognition 

Character-Fields 

14. 

accepted  correctly  recognized  character-fields 

15.  4/{“- 

accepted  incorrectly  recognized  character-fields 

16. 

correctly  rejected  character-field  recognitions 

17- 

incorrectly  rejected  character-field  recognitions 

18. 

missing  characters  due  to  incorrecdy  rejected  char,  field  recognitions 
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Field  Recognition  Cont’d 

Icon-Fields 

19- 

accepted  correctly  recognized  icon-fields 

20. 

accepted  incorrectly  recognized  icon-fields 

21.  Rci%f^ 

rejected  correctly  recognized  icon-fields 

22. 

rejected  incorrectly  recognized  icon-fields 

Character  Recognition 

A /^chrrec 
^'-'char 

accepted  correctly  recognized  characters 

24. 

accepted  incorrectly  recognized  characters 

25. 

missing  characters 

nr'Chrrec 
26-  ^^char 

rejected  correctly  recognized  characters 

0-7  Djchrrec 
^^char 

rejected  incorrectly  recognized  characters 

Figiire  4;  Table  of  scoring  accumulators  and  their  definitions. 
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23  Performance  Measures 


Form-Based 

(1)  FORMl 

(2)  FORM2 

(3)  FORMS 

(4)  FORM4 

fraction  of  all  forms  accepted  and  correctly  identified 
fraction  of  all  forms  not  accepted  and  not  correctly  identified 
fraction  of  accepted  forms  correctly  identified 
fraction  of  accepted  forms  incorrectly  identified 

(5)  FORM5  fraction  of  all  form  identifications  rejected 

Character-Field  Based 

(6)  CHRFLDl  fraction  of  all  character- fields  accepted  and  correctly  recognized 


(7)  CHRFLD2 

(8)  CHRFLD3 

(9)  CHRFLD4 

fraction  of  accepted  character-fields  correctly  recognized 
fraction  of  all  character-fields  missed  due  to  rejected  form  identifications 
fraction  of  all  character- fields  missed  due  to  accepted  incorrect  form 
identifications 

Icon-Field  Based 

(10)  ICOFLDl 

(11)  ICOFLD2 

fraction  of  all  accepted  and  correctly  recognized  icon-fields 
fraction  of  accepted  and  correctly  recognized  icon-fields  from  all 
accepted  icon-field  identifications 

(12)  ICOFLD3 

(13)  ICOFLD4 

fraction  of  all  icon-fields  missed  due  to  rejected  form  identifications 
fraction  of  all  icon-fields  missed  due  to  accepted  incorrect  form 

identifications 

Combined  Field-Based 

(14)  FIELD  1 fraction  of  all  fields  accepted  and  correctly  recognized 


(15)  FIELD2 

(16)  FIELDS 

(17)  FIELD4 

fraction  of  accepted  and  correctly  identified  fields  correctly  recognized 
fraction  of  all  fields  missed  due  to  rejected  form  identifications 
fraction  of  all  fields  missed  due  to  accepted  incorrect  form 
identifications 

Character-Based 
(18)  CHARI 

fraction  of  correctly  recognized  characters  including  characters  missed 
due  to  rejection 

(19)  CHAR2 

(20)  CHARS 

(21)  CHAR4 

(22)  CHAR5 

(23)  CHAR6 

(24)  CHAR7 

(25)  CHAR8 

(26)  CHAR9 

fraction  of  correctly  recognized  characters  from  all  accepted  field  recognitions 

fraction  of  accepted  correctly  recognized  characters 

fraction  of  all  character  recognitions  rejected 

fraction  of  characters  rejected  from  all  accepted  field  recognitions 

fraction  of  correctly  recognized  characters  rejected 

fraction  of  characters  missed  due  to  rejected  form  identifications 

fraction  of  all  accepted  and  correctly  recognized  characters 

fraction  of  accepted  and  correctly  recognized  characters  from  all  accepted 

field  recognitions 

(27)  CHARIO 

fraction  of  aU  characters  missed  due  to  accepted  incorrect  field 
identifications 

Figure  5:  Table  of  recognition  system  performance  measures  and  their  definitions. 
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23.1  Form-Based  Performance  Measures 


FORM!  = 


4 f^frmid 
^ ^form 


total 


form 


(1) 


FORM!  = 


A jfrmid  , nfrmid 


total 


form 


(2) 


FORMS  = 


A 

^ ^form 


Arfrmid 

^'-form 


A jfrmid 
^ form 


(3) 


FORM4  = 


A jfrmid 
^rform 


A + A rFmid 

^^form  ^^^form 


(4) 


FORMS  = 


nfrmid 

^form 


total 


form 


(5) 


232  Character-Field  Based  Performance  Measures 


CHRFLDl  = 


A jr-fldrec 
^^chrfld 


total 


chrfld 


(6) 


CHRFLD2  = 


A j^fldrec 
chrfld 


A j^fldrec  , A jfldrec 
chrfld  ^^^chrfld 


(7) 
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CHRFLD3  = 


total  ^hrfld 


(8) 


CHRFLD4  = 


total 


chrfld 


(9) 


233  Icon-Field  Based  Performance  Measures 


ICOFLDl  = 


A pfldrec 


total; 


icofld 


(10) 


ICOFLDl  = 


. ^fldrec 
icofld 


A r'fldrec  . a jfldrec  . n/^fldrec  . n jfldrec 
^ ^ icofld  ^ ^ ^icofld  icofld  ^Ucofld 


(11) 


ICOFLD3  = 


pj^rmid 

icofld 

^^^^hcoRd 


(12) 


ICOFLDA  = 


^^^^hcofld 


(13) 


23.4  Combined  Field-Based  Performance  Measures 


FIELD!  = 


'chrfld 


icofld 


^^^^^chrfld  ^^^^hcofld 


(14) 
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FIELD!  = 


FIELDS  = 


total ^hrfld  ^^^^hcofld 


(16) 


FIELDA  = 


^^^chrfld  ^ ^^^icofld 

total chrf Id  ^ ^^^^hcofld 


(17) 


2J^  Character-Based  Performance  Measures 


CHARI  = 


AC 


chrrec 

char 


+ RC 


chrrec 

char 


(18) 


CHAR2  = 


AC 


chrrec 

char 


+ RC 


chrrec 

char 


AC 


chrrec 

char 


+ AI 


chrrec 

char 


+ RC 


chrrec 

char 


+ RI 


chrrec 

char 


(19) 


CHAR3  = 


AC 


chrrec 

char 


A ^chrrec  , a jchrrec 
^'-char  '^^Hhar 


(20) 


CHAR^  = 


RC 


chrrec 

char 


+ RI 


chrrec 

char 


total  ref chr 


(21) 


* Note  that  FIELD2  was  incorrect  in  NISTTR  5129.  The  corrections  have  been  included  in  this  cross-reference. 
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n^chrrec  , p jchrrec 
^^char  ~^^^char 


CHARS  = 


AC 


chrrec 

char 


+ AI 


chrrec 

char 


+ RC 


chrrec 

char 


+ RI 


chrrec 

char 


(22) 


CHAR6 


RC 


chrrec 

char 


AC 


chrrec 

char 


+ RC 


chrrec 

char 


(23) 


CHARI  = 


R^Z'f 


total 


refchr 


(24) 


CHARS  = 


AC 


chrrec 

char 


tOtalrefchr 


(25) 


CHAR9 


AC 


chrrec 

char 


AC 


chrrec 

char 


+ AI 


chrrec 

char 


+ RC 


chrrec 

char 


+ RI 


chrrec 

char 


+ M 


chrrec 

char 


(26)* 


CHARIO  = 


^^char 


total 


refchr 


(27) 


* Note  that  CHAR9  was  incorrect  in  NISTER.  5129.  The  corrections  have  been  included  in  this  cross-reference. 
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3.  Scoring  Package  Output  Files 

The  Scoring  Package  generates  two  output  files  defined  in  the  User’s  Guide  (NISTIR  4950).  The  first  file,  a siunmary  report,  lists 
the  performance  measures  computed  by  the  Scoring  Package,  and  the  second  file,  a fact  sheet,  gives  a detailed  accounting  of  var- 
ious event  statistics.  The  summary  report  is  documented  in  Section  3.1,  and  the  fact  sheet  is  documented  in  Section  3 .2.  Comments 
delimited  by  “<”  and  “>”  have  been  added  to  the  file  listings  for  the  purpose  of  documentation.  Where  appropriate,  the  statistics 
reported  in  these  two  files  have  been  cross-referenced  to  scoring  accumulators  and  perfcsmance  measures  defined  in  the  Methods 
Paper  (NISTIR  5129). 
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3.1  Summary  Report  Description 

Summary:  < beginning  of  summary  report  > 

TOTALS  ( output=FCItdA,of=foiTn.sum,cf=form.fct ) < scoring  profile  options  selected  > 


Draft  standard  measures: 


< fundamental  accumulators : 

> 

< 

> 

< 

> 

< 

rt  = /?c^^7^ 

> 

< 

> 

< 

RM  = 

> 

Accumulators:  TP=1648  FP=43  M=36  RT=45  RF=18  RM=164 
Character  recognition  decision: 

< CHARI  (18)  > 

accuracy:  88.8410%  ( 1648/1855  ) 

< CHAR2  (19)  > 

accuracy  (form  right):  97.4571%  ( 1648/1691) 
Character  ouQ^ut: 

< CHAR3  (20)  > 

accuracy:  98.4644%  ( 1603  / 1628  ) 

Field  accuracy: 

< HELDl  (14)  > 

accuracy  (including  icons):  81.2762%  (777/956) 

Character  rejection  rates: 

< CHAR4  (21)  > 

aU:  3.3475%  (63/1882) 

< CHARS  (22)  > 

all  hypotheses:  3.7256%  (63/1691  ) 

< CHAR6  (23)  > 

matches:  2.7306%  (45/1648) 

< percentage  of  substituted  characters  rejected  > 

substitutions:  44.1176%  (15/34) 

< percentage  of  inserted  characters  rejected  > 

insertions:  33.3333%  ( 3 / 9 ) 

< CHAR7  (24)  > 

: all  (due  to  form  type):  8.7141%  ( 164/1882) 

Fields  (excluding  icons): 

< CHRFLDl  (6)  > 

: accuracy:  81.7010%  (634/776) 

< CHRFLD2  (7)  > 

accuracy  (with  form  right):  90.1849%  ( 634  / 703  ) 

< CHRFLD3  (8)  > 

rejected  (due  to  form  type):  9.4072%  ( 73  / 776 ) 

< CHRFLD4  (9)  > 

deleted  (due  to  form  wrong):  0.0(X)0%  ( 0 / 776 ) 
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Fields  (including  icons): 

< FIELDl  (14)  > 

accuracy:  81.2762%  {111  1956) 

< FIELD2  (15)  > 

accuracy  (with  form  right):  89.8266%  ( 111  / 865  ) 

< HELDS  (16)  > 

rejected  (due  to  form  type):  9.5188%  (91/956) 

< nELD4  (17)  > 

deleted  (due  to  form  wrong):  0.0(X)0%  ( 0 / 956 ) 

Characters: 

< CHARS  (25)  > 

accuracy:  85.1753%  ( 1603/1882) 

< CHAR9  (26)  > 

accuracy  (with  form  right):  94.7960%  ( 1603  / 1691 ) 

< CHAR7  (24)  > 

: rejected  (due  to  form  type):  8.7141%  ( 164  / 1882 ) 

< CHAR  10  (27)  > 

: deleted  (due  to  form  wrong):  0.0000%  ( 0 / 1882 ) 

Icons: 

<ICOFLD1(10)> 

accuracy:  79.4444%  ( 143  / 180 ) 

< ICOFLD2  (11)  > 

accuracy  (with  form  right):  88.2716%  (143/162) 

< ICOFLD3  (12)  > 

rejected  (due  to  form  type):  10.0000%  ( 18  / 180 ) 

< ICOFLD4  (13)  > 

deleted  (due  to  form  wrong):  0.0000%  ( 0 / 180 ) 

Form  type  identification: 

< FORMl  (1)  > 

accuracy:  90.9091%  (10/11) 

< FORM2  (2)  > 

failure  rate:  9.0909%  ( 1 / 11 ) 

< FORM3  (3)  > 

accuracy  (excluding  rejected):  100.0000%  ( 10  / 10 ) 

< FORM4  (4)  > 

: failure  rate  (excluding  rejected):  0.0000%  ( 0 / 10 ) 

< FORM5  (5)  > 

rejected:  9.0909%  ( 1 / 11 ) 
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32  Fact  Sheet  Description 


form  type: 

< form-level  accumulators  > 

cxmnt:  11 

< totalf > 

form 

rejected:  1 

not  rejected,  right:  10 

not  rejected,  wrong:  0 

< > 
form 

icon  fields: 

< icon  field  accumulators  > 

count:  180 

< ^O^^hcofld  > 

< indented  counts  are  subsets  of  all  icon  fields  scored  > 

form  type  rejected:  18 

form  type  wrong  and  not  rejected:  0 

form  type  right  and  not  rejected:  162 

< indented  counts  are  subsets  of  aU  forms  correctly  identified  and  not  rejected  > 

right:  143 

o'  O'  > 

wrong:  19 

rejected:  15 

A 

+ 

(< 

V 

not  rejected:  147 

. Arf^drec  .ffldrec 
^ ^^icofld^^Ucofld  > 

matches;  157 

^ Arf^drec  T^f^fldrec 

< indented  counts  are  subsets  of  all  correct  icon  fields  ignoring  rejection  > 

rejected:  14 

- Df-fldrec 

not  rejected:  143 

A 

I:*  ■'5 

V 

mismatches;  5 

< indented  counts  are  subsets  of  all  incorrect  icon  fields  ignoring  rejection  > 

rejected:  1 

A 

V 

not  rejected:  4 

< ■'O  > 

not  present  / not  found:  115 

< # of  empty  icon  fields  detected  correctly  > 

not  present  / founi  3 

< # of  empty  icon  fields  detected  incorrectly  > 

present  / not  found:  2 

< # of  non-empty  icon  fields  detected  incorrectly 

present  / found:  42 

< # of  non-empty  icon  fields  detected  correctly  > 
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character  fields: 


< character  field  accumulators  > 


count;  776  < > 

< indented  counts  are  subsets  of  all  character  fields  scored  > 


form  type  rejected:  73  < ^ 

form  type  wrong  and  not  rejected:  0 < > 

form  type  right  and  not  rejected:  703  <ACfchrfid  + > 

< indented  counts  are  subsets  of  all  character  fields  on  forms  correctly  identified  and  not  rejected  > 


right:  634 
wrong:  69 


chrfld 


characters;  < character-level  accumulators  > 

< indented  counts  are  subsets  of  all  character  scored  > 
in  alignments:  1891  < # of  character  alignment  positions  > 

hypothesis;  1691  < # of  hypothesized  characters  > 

reference:  1882  < > 

< indented  counts  are  subsets  of  all  reference  characters  scored  > 
form  type  rejected:  164  < ^ 

form  type  wrong  and  not  rejected:  0 < > 

form  type  right  and  not  rejected;  1691  < > 

< indented  counts  are  subsets  of  all  reference  characters  on  forms  correctly  identified  and  not  rejected  > 


:RC 


•chrrec 


char 


■i-RIi 


hrrec 

char 


<ACitT+Rc:tT> 


rejected:  63 
not  rejected;  1628 
correct:  1648 

< indented  counts  are  subsets  of  all  correct  characters  ignoring  rejection  > 

rejected:  45  < RC‘^Jh^^‘^  > 

not  rejected:  1603  < > 

substitutions:  34  < # of  substituted  characters  ignoring  rejection  > 

< indented  counts  are  subsets  of  all  substituted  characters  ignoring  rejection  > 

rejected:  15  < # of  substituted  characters  rejected  > 

not  rejected:  19  < # of  substituted  characters  accepted  > 

insertions:  9 < # of  inserted  characters  ignoring  rejection  > 

< indented  counts  are  subsets  of  all  inserted  characters  ignoring  rejection  > 

rejected:  3 < # of  inserted  characters  rejected  > 

not  rejected:  6 < # of  inserted  characters  accepted  > 

deletions;  36  < > 


Accumulators;  TP=1648  FP=43  M=36  RT=45  RF=18  RM=164  < fundamental  accumulators  > 
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